Parallel-hierarchical model for machine comprehension on small data

ABSTRACT

Examples of the present disclosure provide systems and methods relating to a machine comprehension test with a learning-based approach, harnessing neural networks arranged in a parallel hierarchy. This parallel hierarchy enables the model to compare the passage, question, and answer from a variety of perspectives, as opposed to using a manually designed set of features. Perspectives may range from the word level to sentence fragments to sequences of sentences, and networks operate on word-embedding representations of text. A training methodology for small data is also provided.

PRIORITY

This application claims the benefit of, and priority to, U.S.Provisional Application No. 62/309,139, entitled, “Parallel-HierarchicalModel for Machine Comprehension on Small Data,” filed on Mar. 16, 2016,which is hereby incorporated by reference in its entirety.

FIELD

The present specification relates processing comprehension of text.

BACKGROUND

Understanding unstructured text is a major goal within natural languageprocessing. machine comprehension tests (e.g., MCTest) pose questionsbased on short text passages to evaluate such understanding.Comprehension systems aim to extract the correct answer from the passagewhich contains it. Partly because of its limited size, prior work onMCTest has focused mainly on feature-based approaches.

It is with respect to these and other general considerations that theaspects disclosed herein have been made. Also, although relativelyspecific problems may be discussed, it should be understood that theexamples should not be limited to solving the specific problemsidentified in the background or elsewhere in this disclosure.

SUMMARY

A machine comprehension system with a learning-based approach,harnessing neural networks arranged in a parallel hierarchy is provided.This parallel hierarchy enables the model to compare the passage,question, and answer from a variety of perspectives, as opposed to usinga manually designed set of features. Perspectives may range from theword level to sentence fragments to sequences of sentences, and networksoperate on word-embedding representations of text. A trainingmethodology for small data is also provided.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Additionalaspects, features, and/or advantages of examples will be set forth inpart in the description which follows and, in part, will be apparentfrom the description, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive examples are described with reference tothe following figures.

FIG. 1 illustrates an overview of an example method for performingparallel-hierarchical machine comprehension on a data set.

FIG. 2 illustrates an example process for performingparallel-hierarchical machine comprehension on a data set using semanticand word-by-word processes in parallel.

FIG. 3 is a block diagram illustrating example physical components of acomputing device with which aspects of the disclosure may be practiced.

FIGS. 4A and 4B are simplified block diagrams of a mobile computingdevice with which aspects of the present disclosure may be practiced.

FIG. 5 is a simplified block diagram of a distributed computing systemin which aspects of the present disclosure may be practiced.

FIG. 6 illustrates a tablet computing device for executing one or moreaspects of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various aspects of the disclosure are described more fully below withreference to the accompanying drawings, which form a part hereof, andwhich show specific exemplary aspects. However, different aspects of thedisclosure may be implemented in many different forms and should not beconstrued as limited to the aspects set forth herein; rather, theseaspects are provided so that this disclosure will be thorough andcomplete, and will fully convey the scope of the aspects to thoseskilled in the art. Aspects may be practiced as methods, systems ordevices. Accordingly, aspects may take the form of a hardwareimplementation, an entirely software implementation or an implementationcombining software and hardware aspects. The following detaileddescription is, therefore, not to be taken in a limiting sense.

Humans learn in a variety of ways. For instance, humans learn bycommunication with each other, and by study—the reading of text.Comprehension of unstructured text by machines, at a near-human level,is one goal within natural language processing. It has garneredsignificant attention from the machine learning research community inrecent years, giving rise to a host of models and datasets on which totest them.

Typically, machine comprehension (MC) is evaluated by posing a set ofreading questions based on a text passage. This is akin to the readingtests we all took in school. Such tests are objectively gradable, andcan be used to assess a range of abilities from basic understanding tocausal reasoning to inference. Given a text passage and a question aboutits content, a machine is tested on its ability to determine the correctanswer to the question. MCTest is a small but complex benchmark thatrequires not only extraction but also inference and limited reasoning.

Aspects of the present disclosure generally relate to aparallel-hierarchical approach to machine comprehension process that isgeared towards small data. For example, MCTest, comprising only 660passages with related questions and multiple-choice answer candidates,is an example of such a data set. For ease of disclosure, the variousaspects described herein may be described with respect to processingdata related to MCTest. One of skill in the art will appreciate that thevarious aspects disclosed herein, however, may be performed using othertypes of input. For example, various types of documents, communicationsusing natural language expressions, books, web pages, log files, or anytype of electronic document may be processed using the various aspectsdisclosed herein.

In various aspects, the model compares question and answer to text usingseveral distinct perspectives. As used herein, a question combined withone of its answer candidates is referred to as a hypothesis, which willbe described in further detailed below. One exemplary perspective, thesemantic perspective, compares the hypothesis to sentences in the textviewed as single, self-contained thoughts which may be represented usinga sum and transformation of word embedding vectors. Another exemplaryperspective, the word-by-word perspective, focuses on similarity matchesbetween individual words from hypothesis and text, at various scales. Asin the case with the semantic perspective, there may be a sententialscale. In another aspect, a sliding Gaussian window acting on asubsentential scale, which implicitly considers the linear distancebetween matched words, may also be used. Such a word-level slidingwindow operates on two different views of text sentences such as thesequential view, where words appear in their natural order, and thedependency view, where words may be reordered based on a linearizationof the sentence's dependency graph. These distinct perspectivesnaturally form a hierarchy of processing, which will be described infurther detail with respect to FIG. 2. In various aspects, words may berepresented within the model by embedding vectors.

With the exception of the dependency-graph linearization (which relieson an eigendecomposition), the end-to-end comparison function thatimplements and synthesizes these perspectives may be a differentiableneural model, trainable by backpropagation. Specific feature engineeringis not performed (although the dependency graph can be considered anadditional feature), and the model takes as input only word embeddings.This is in contrast to most previous efforts on MCTest that use numerousfeatures. In experiments, the model achieves state-of-the-art accuracyon MCTest, outperforming all known alternatives on the test set.

To facilitate learning on small data, a training scheme may be provided.The model's neural networks may be initialized to perform specificheuristic functions. These were functions may be used in order to yieldsatisfactory performance on the dataset. In doing so, the trainingapproach gives the model a safe, reasonable baseline from which to startlearning.

It is to be appreciated by a person of skill in the art with the benefitof this description that machine comprehension requires machines toanswer questions based on unstructured text. This can be viewed asselecting the best answer from a set of candidates. In themultiple-choice case, candidate answers are predefined by the testdesigner, but candidate answers may also be undefined yet restricted(e.g., to yes, no, or any noun phrase in the text).

For each question q, let T be the unstructured text and A={a₁; . . . ,a_(m)} be the set of candidate answers to q. The machine comprehensiontask reduces to selecting the answer â_(i) that has the highest evidencegiven T. An answer and a question are combined into a hypothesis,h_(i)=f(q; a_(i)). To facilitate comparisons of the text with thehypotheses, the passage is broken down into sentences t_(j), T {t_(j)}.In this setting, q, a_(i), and t_(j) each represent a sequence ofembedding vectors, one for each word and punctuation mark in therespective item.

In addition, is to be appreciated by a person of skill in the art withthe benefit of this description that models may be divided into a numberof different approaches, such as a feature-based approach and alearning-based approach. The feature-based approach relies on manuallyengineered features of the data that require significant effort on thepart of a human designer, and/or significant auxiliary tools to extractthese features for use. The learning-based approach typically harnessesa single feature such as, for example, vector-representations of words,or word embeddings. These may be pretrained and fine-tuned, or trainedentirely on the dataset. Typically, word embeddings may be input into adeep neural network which processes and compares text to questions andanswers. Among deep-learning models, mechanisms of attention and workingmemory may be used.

Feature-Based Models

MCTest can be treated as a structured prediction problem, searching fora latent answer entailing structure connecting question, answer, andtext. This structure corresponds to the best latent alignment of ahypothesis with appropriate snippets of the text. The process of(latently) selecting text snippets may be related to the attentionmechanisms typically used in the deep statistical learning models formachine comprehension and machine translation. The model uses event andentity co-reference links across sentences along with a host of otherfeatures. These features may include specifically trained word vectorsfor synonymy; antonymy and class-inclusion relations from externaldatabase sources and/or dependencies and semantic role labels. The modelmay be trained using a latent structural support vector machine (SVM)extended to a multitask setting, so that questions are first classifiedusing a pretrained top-level classifier. This enables the system to usedifferent processing strategies for different question categories. Themodel also combines question and answer into a well formed statementusing rules.

In some embodiments, the model can be simplified in terms of thefeatures it has access to, the training procedure (e.g., stochasticgradient descent, alternate minimization, etc.), questionclassification, and question-answer combination (e.g., concatenation,mean, a set of rules etc.).

It is to be appreciated that the baseline feature set may be combinedwith feature modules for a co-reference resolution, stemming, and rulesfor question classification. The simple model improved on the originalbaseline, but does not approach state-of-the-art results.

The baseline feature may also be augmented with features for syntax,frame semantics, coreference, and/or word embeddings. The frame semanticfeatures in particular may require significant manual effort. Featuresmay be combined using a linear latent-variable classifier trained tominimize a max-margin loss function. Questions and answers may becombined using a set of rules.

Learning-Based Models

While the learning-based approach may be used, this approach isgenerally less accurate than the feature-based model described above onMCTest. This is because the dataset is small and complex. A neuralnetwork that captures complex patterns typically requires manyparameters, and the more parameters, the more training data is requiredto tune them all. Deep models develop their own features, rather thanbeing provided them by a designer, but this is a data-intensive process.

Deep learning approaches on MCTest have also been carried out. Theymeasured the performance of the Attentive Reader and the NeuralReasoner, both recurrent models with attention mechanisms, and alsodeveloped an attention-based convolutional network, the HABCNN. Theirnetwork operates on a hierarchy, providing further evidence of thepromise of hierarchical perspectives. Specifically, the HABCNN processestext at the sentence level and the snippet level, where the lattercombines adjacent sentences like the n-gram input. Embedding vectors forthe question and the answer candidates are combined and encoded viaconvolutional network. This encoding modulates attention over sentenceand snippet encodings, followed by maxpooling to determine the bestmatches between question, answer, and text.

Parallel-Hierarchical Model

FIG. 1 illustrates an overview of an example method 100 for performingparallel-hierarchical machine comprehension on a data set. Flow beginsat operation 102 where a data set is received. In one aspect, thereceived data set may include natural language text. In one example, thereceived data set includes one or more questions, one or more answers,and one or more textual entries which may be analyzed to select theappropriate answer to a question. Flow continues to operation 104 wherethe received text is converted to a distributed representation. In oneexample, converting the text to a distributed representation maycomprise determining a vector representation for each word and/or groupof words in the received data set. In one aspect, a neural network maybe employed convert the text to a distributed representation. One suchneural network may be a multilayer perceptron (MLP). One of skill in theart, however, will appreciate that other types of neural networks may beused without departing from the spirit of this disclosure.

Flow continues to operation 106 where the distributed representation ofthe received text may be analyzed in parallel in order to determine thecorrect answer to the one or more questions. In examples, variousdifferent processes may be employed to analyze the distributed text atoperation 106 including, but not limited to, a semantic perspectiveprocess, a word-by-word perspective process, a sequential perspectiveprocess, a sliding window processes, and the like. While specificexamples different types of processes performed in parallel have beendescribed herein, on of skill in the art will appreciate that othertypes of processes may be employed without departing from the scope ofthis disclosure.

Flow continues to operation 108 where the output form the differentprocesses may be combined into a single result set. In one example, thecombination of the results may be performed by a neural network. Forexample, the outputs from the different processes may be provided to anMLP. In addition to combining the results, the neural network may befurther operable to order the hypotheses by likelihood of correctnessbased upon the combined results of the various processors employed atoperation 108.

In some aspects, the results may be further refined. For example,further processing may be performed on ordered hypotheses generated bycombining the results at operation 108. If refinement is to beperformed, flow branches YES from decision operation 110 to operation112. At operation 112, a predetermined number of the top results may beselected for further processing. The number of results selected may bedetermined in advanced by the process performing the method 100 or maybe determined dynamically based upon the combined results generated atoperation 108. Flow may then return to operation 106 where a number oftop results may be provided as input to the processes executing inparallel at operation 106.

Returning to operation 110, in alternate aspects, the refinementprocessing may not be performed or may not be necessary. In suchaspects, flow branches NO to operation 114 where a top result may beprovide. In certain aspects, more than one top rated result mays bereturned to operation 114.

FIG. 2 illustrates an example process for performingparallel-hierarchical machine comprehension on a data set using semanticand word-by-word processes in parallel. In this model, each of theperspectives (semantic perspective and word-by-word perspective) arediscussed separately, then describe how they are combined. The differentperspectives use word embedding vectors as input. Below, subscripts areused to index elements of sequences, like word vectors, and superscriptsto indicate whether elements come from the input data 202 which mayinclude text data 202A, question data 202B, and/or answer data 202B. Inparticular, subscripts (k; m; n; p) are used to index sequences of wordsfrom the text, question, answer, and hypothesis, respectively. In oneexample, the text data may be parsed by individual words, as a unigram(e.g., a single sentence), a bigram (e.g., two sentences), a trigram(e.g., three sentences) or any number of combined sentences.

In the semantic perspective process 208, each sentence of the text maybe a sequence of d-dimensional word vectors: t_(j)={x_(k) ^(t)}; x_(k)^(t)∈

^(d). The semantic vector st may be computed using a MLP plus sumprocess 204. For example, the semantic vector st may be computed byembedding the word vectors into a D-dimensional space using a multilayerperceptron (MLP) whose inner transformation is augmented with a sum;i.e.,

$\begin{matrix}{s^{t} = {{f\left( {{A^{t}{\sum\limits_{k}{\omega_{k}x_{k}^{t}}}} + b_{A}^{t}} \right)}.}} & (1)\end{matrix}$

The matrix A^(t)∈

^(D×d) the bias vector b^(t) _(A)∈

^(D) and in certain aspects, for f the leaky ReLU function may be used.The scalar ω_(k) may be a trainable weight associated to each word inthe vocabulary. These scalar weights form an exogenous or bottom-upattention that depends only on the input stimulus. They scaler weightsmay, for example, learn to perform the function of stopword lists in asoft, trainable way, to nullify the contribution of unimportant fillerwords.

The semantic representation of a hypothesis may be formed analogously,except that the question word vectors x^(q) _(m) and answer word vectorsx^(a) _(n) are combined as a single sequence {x_(p) ^(h)}={x_(m) ^(q),x_(n) ^(a)}. For semantic vector s_(h) of the hypothesis, a uniquetransformation matrix A^(h)∈

^(D×d) and bias vector b^(h) _(A)∈

^(D) may be used.

These transformations map a text sentence and a hypothesis into a commonspace where they can be compared. The semantic match between textsentence and hypothesis may be computed using the cosine similarity,M ^(sem)=cos(s ^(t) ,s ^(h)).  (2)

The first step in building the word-by-word perspective 210 is totransform word vectors from a text sentence 202A, question 202B, andanswer 202C through one or more respective MLPs 206. That is,w^(y)=f(B^(y)x^(y)+{hacek over (b)}^(y) _(B)) where B^(y)∈

^(D×d), b^(y) _(B)∈

^(D), and y∈{t,q,a}. In contrast with the semantic perspective 208, thequestion and answer candidates are kept separate in the word-by-wordperspective 210. This is because matches to answer words are inherentlymore important than matches to question words, and the model learns andutilizes this. Various different word-by-word perspective processes maybe employed to evaluate hypotheses, such as a sentential process 210A, asliding window (SW) sequential process 210B, and/or a SW-dependencyprocess 210C.

In the present aspect, matches between hypotheses and text sentences atthe word level are computed using the sentential process 210A. Thiscomputation may be split over questions and answers, and as before usesthe cosine similarity:c _(km) ^(q)=cos(w _(k) ^(t) ,w _(m) ^(q)),  (3)c _(kn) ^(a)=cos(w _(k) ^(t) ,w _(n) ^(a)).  (4)

The word-by-word match between a text sentence and question may bedetermined by taking the maximum over k (finding the text word thatmatches each question word) and then taking a weighted mean over m(finding the average match over the full question):

$\begin{matrix}{M^{q} = {\frac{1}{Z}{\sum\limits_{m}{\omega_{m}{\max\limits_{k}{c_{km}^{q}.}}}}}} & (5)\end{matrix}$

Here, ω_(m) is the word weight and Z normalizes the weights to sum tounity over the question. The match between a sentence and answercandidate, M^(a), is defined analogously. Finally, the matches arecombined to question and answer according toM ^(word)=α₁ M ^(q)+α₂ M ^(a)+α₃ M ^(q) M ^(a).  (6).

Here, the α are trainable parameters that control the relativeimportance of the different components.

The sequential sliding window process 210B hearkens back to the originalMCTest baseline, with two key differences. First, a soft match betweenword vectors may be used rather than an exact lexical match of words;second, the sliding window decays from its focus word according to aGaussian distribution, which may be manifest as weight for eachposition. This latter modification enables the window to use informationabout the distance between word matches; the original baseline made useof distance information through a predefined function.

The sliding window scans over the words of the text as one continuoussequence, without sentence breaks. Each window is treated like asentence in the previous subsection, but a location based weight γ_(r)is included, where r indexes the position in the window. The cosinesimilarity is adapted asc _(km,r) ^(y)=γ_(r) cos(w _(k) ^(t) ,w _(m) ^(y)),y∈{q,a}.  (7)

The location weights, initialized with a Gaussian, can be fine-tuned intraining. The final matching score, denoted as M^(sws), is computed asin (5) and (6).

The dependency sliding window process 210B operates identically to thelinear sliding window, but on a different view of the text passage. Thepassage may be divided into sentences. Then the a dependency parser maybe applied to each text sentence to build a dependency graph. Oneexample of a dependency parser that may be employed is the StanfordDependency Parser, however other dependency parsers may be sued withoutdeparting from the spirit of this disclosure. This graph has n_(w)vertices, one for each word in the sentence. From the dependency graphthe Laplacian matrix L∈{circumflex over ( )}

^(n) ^(w) ^(×n) ^(w) is formed and determine its eigenvectors.

The second eigenvector u₂ of the Laplacian is known as the Fiedlervector. It is the solution to the minimization

$\begin{matrix}{{\min\limits_{g}{\sum\limits_{i,{j = 1}}^{N}{\eta_{ij}\left( {{g\left( v_{i} \right)} - {g\left( v_{j} \right)}} \right)}^{2}}},} & (8)\end{matrix}$

where v_(i) are the vertices of the graph, and n_(ij) is the weight ofthe edge from vertex i to vertex j. Thus, the Fiedler vector maps aweighted graph onto a line such that connected nodes stay close,modulated by the connection weights. This enables the reordering of thewords of a sentence based on their proximity in the dependency graph.The reordering of the words is given by the ordered index setI=arg sort(u ₂).  (9)

To give an example of how this works, consider the following sentencefrom MCTest and its dependency-based reordering: the sentence “Jenny,Mrs. Mustard's helper, called the police.” May be reordered as “thepolice, called Jenny helper, Mrs. 's Mustard.”

Sliding-window-based matching on the original sentence will answer thequestion “Who called the police?” with “Mrs. Mustard.” The dependencyreordering enables the model to determine the correct answer.

The dependency sliding window process 210C may uses an auxiliary tool,such as the Stanford Dependency Parser. These are rearranged based onthe linearized dependency graph. The output of the dependency slidingwindow component is M_(swd) and is formed analogously to M_(sws).

Some data sets, such as the MCTest, may include questions where theevidence for an answer spans several sentences. The motivation here wasto ensure the task could not be solved by lexical techniques alone, butwould instead require some form of inference or limited reasoning. Inorder to deal with such examples, the model also takes in n-grams ofsentences, i.e., sentence pairs and triples strung together. The modeltreats these exactly as it does single sentences, applying all functionsdetailed above. A later pooling operation combines scores across alln-grams (including the single-sentence input).

With n-grams, the model can synthesize information distributed acrosscontiguous sentences. In some cases, however, the required evidence isspread across distant sentences. To give the model some capacity to dealwith this scenario, the top N sentences as scored by all the precedingfunctions is used, and then repeat the scoring computations viewingthese top N as a single sentence.

Another neural network 212, such as an MLP, is used to combine M^(sem),M^(word), M^(swd), and M^(sws) as a final matching score M_(i) 216 foreach answer candidate. This neural network also pools and combines theseparate n-gram scores, and has a linear activation function.

Before training, the neural-network components of the model areinitialized to perform sensible heuristic functions using a trainingwheel approach. It was found empirically that above 50% accuracy can beachieved on MCTest using a simple sum of word vectors followed by a dotproduct between the question sum and the hypothesis sum. Therefore, theMLP for the semantic perspective is initialized to perform a sum of wordvectors. This is done by initializing A^(y) as the identity matrix andb^(y) _(A) as the zero vector, y∈{t,h}. Recall that the activationfunction is a ReLU so that positive outputs are unchanged.

The word-by-word MLPs were initialized with identity matrices and0-biases as well, while the MLP for perspective combination wasinitialized to perform a sum of individual scores, using a zerobias-vector and a weight matrix of ones.

Training

It is to be appreciated that determining correct answers is challengingif a dataset is both complicated and small. Its size limits the numberof parameters that can be trained, and prevents learning any complexlanguage modeling simultaneously with the capacity to answer questions.However, there are many use-cases in which comprehension over limiteddata would be handy (e.g., user manuals, internal documentation, etc.),and work towards more efficient learning from limited data is importantin its own right.

In this section details of an exemplary training procedure aredescribed. The training objective is to minimize the ranking loss

$\begin{matrix}{{\left( {T,q,A} \right) = {\max\left( {0,{\mu + {\max\limits_{i}M_{i \neq *}} - M_{*}}} \right)}},} & (10)\end{matrix}$

where μ is a constant margin and the maximum taken over i so thatranking the correct answer over the best-ranked incorrect answer (ofwhich there may be multiple). This approach worked better than comparingthe correct answer to the incorrect answers individually.

For word vectors, publicly available embeddings may be used and trained,for example, by example using word2vec on a large corpus. These vectorsare kept fixed throughout training, since it is found that training themwas not helpful (likely because of the size of small data sets). Thevectors may be multidimensional, for example, 300-dimensional (d=300)vectors may be used.

In certain aspects, a stopword list may not used for the text passage.Instead these aspects may rely on the trainable word weights to ascribeglobal importance ratings to words. These weights may be initializedwith the inverse document frequency (IDF) statistic computed over anexemplary corpus. However, a short stopword list may be used forquestions. This list nullifies query words such as {Who, what, when,where, how}, along with conjugations of the verbs “to do” and “to be.”

A heuristic may be used to improve performance on negation questions.When a question contains the words “which” and “not” or “n't,” thehypothesis ranking scores may be negated so that the minimum becomes themaximum.

FIGS. 3-6 and the associated descriptions provide a discussion of avariety of operating environments in which aspects of the disclosure maybe practiced. However, the devices and systems illustrated and discussedwith respect to FIGS. 3-6 are for purposes of example and illustrationand are not limiting of a vast number of computing device configurationsthat may be utilized for practicing aspects of the disclosure, describedherein.

FIG. 3 is a block diagram illustrating physical components (e.g.,hardware) of a computing device 300 with which aspects of the disclosuremay be practiced. The computing device components described below may besuitable for the computing devices described above. In a basicconfiguration, the computing device 300 may include at least oneprocessing unit 302 and a system memory 304. Depending on theconfiguration and type of computing device, the system memory 304 maycomprise, but is not limited to, volatile storage (e.g., random accessmemory), non-volatile storage (e.g., read-only memory), flash memory, orany combination of such memories. The system memory 304 may include anoperating system 305 and one or more program modules 306 suitable forperforming the various aspects disclosed herein such as contextassociation generation component 324 and isolated collection adaptationcomponent 326. The operating system 305, for example, may be suitablefor controlling the operation of the computing device 300. Furthermore,embodiments of the disclosure may be practiced in conjunction with agraphics library, other operating systems, or any other applicationprogram and is not limited to any particular application or system. Thisbasic configuration is illustrated in FIG. 3 by those components withina dashed line 308. The computing device 300 may have additional featuresor functionality. For example, the computing device 300 may also includeadditional data storage devices (removable and/or non-removable) suchas, for example, magnetic disks, optical disks, or tape. Such additionalstorage is illustrated in FIG. 3 by a removable storage device 309 and anon-removable storage device 310.

As stated above, a number of program modules and data files may bestored in the system memory 304. While executing on the processing unit302, the program modules 306 (e.g., Parallel-hierarchical machinecomprehension process 320) may perform processes including, but notlimited to, the aspects, as described herein. Other program modules thatmay be used in accordance with aspects of the present disclosure mayinclude electronic mail and contacts applications, word processingapplications, spreadsheet applications, database applications, slidepresentation applications, drawing or computer-aided applicationprograms, etc.

Furthermore, embodiments of the disclosure may be practiced in anelectrical circuit comprising discrete electronic elements, packaged orintegrated electronic chips containing logic gates, a circuit utilizinga microprocessor, or on a single chip containing electronic elements ormicroprocessors. For example, embodiments of the disclosure may bepracticed via a system-on-a-chip (SOC) where each or many of thecomponents illustrated in FIG. 3 may be integrated onto a singleintegrated circuit. Such an SOC device may include one or moreprocessing units, graphics units, communications units, systemvirtualization units and various application functionality all of whichare integrated (or “burned”) onto the chip substrate as a singleintegrated circuit. When operating via an SOC, the functionality,described herein, with respect to the capability of client to switchprotocols may be operated via application-specific logic integrated withother components of the computing device 300 on the single integratedcircuit (chip). Embodiments of the disclosure may also be practicedusing other technologies capable of performing logical operations suchas, for example, AND, OR, and NOT, including but not limited tomechanical, optical, fluidic, and quantum technologies. In addition,embodiments of the disclosure may be practiced within a general purposecomputer or in any other circuits or systems.

The computing device 300 may also have one or more input device(s) 312such as a keyboard, a mouse, a pen, a sound or voice input device, atouch or swipe input device, etc. The output device(s) 314 such as adisplay, speakers, a printer, etc. may also be included. Theaforementioned devices are examples and others may be used. Thecomputing device 300 may include one or more communication connections316 allowing communications with other computing devices 350. Examplesof suitable communication connections 316 include, but are not limitedto, radio frequency (RF) transmitter, receiver, and/or transceivercircuitry; universal serial bus (USB), parallel, and/or serial ports.

The term computer readable media as used herein may include computerstorage media. Computer storage media may include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information, such as computer readableinstructions, data structures, or program modules. The system memory304, the removable storage device 309, and the non-removable storagedevice 310 are all computer storage media examples (e.g., memorystorage). Computer storage media may include RAM, ROM, electricallyerasable read-only memory (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other article of manufacturewhich can be used to store information and which can be accessed by thecomputing device 300. Any such computer storage media may be part of thecomputing device 300. Computer storage media does not include a carrierwave or other propagated or modulated data signal.

Communication media may be embodied by computer readable instructions,data structures, program modules, or other data in a modulated datasignal, such as a carrier wave or other transport mechanism, andincludes any information delivery media. The term “modulated datasignal” may describe a signal that has one or more characteristics setor changed in such a manner as to encode information in the signal. Byway of example, and not limitation, communication media may includewired media such as a wired network or direct-wired connection, andwireless media such as acoustic, radio frequency (RF), infrared, andother wireless media.

FIGS. 4A and 4B illustrate a mobile computing device 400, for example, amobile telephone, a smart phone, wearable computer (such as a smartwatch), a tablet computer, a laptop computer, and the like, with whichembodiments of the disclosure may be practiced. In some aspects, theclient may be a mobile computing device. With reference to FIG. 4A, oneaspect of a mobile computing device 400 for implementing the aspects isillustrated. In a basic configuration, the mobile computing device 400is a handheld computer having both input elements and output elements.The mobile computing device 400 typically includes a display 405 and oneor more input buttons 410 that allow the user to enter information intothe mobile computing device 400. The display 405 of the mobile computingdevice 400 may also function as an input device (e.g., a touch screendisplay). If included, an optional side input element 415 allows furtheruser input. The side input element 415 may be a rotary switch, a button,or any other type of manual input element. In alternative aspects,mobile computing device 400 may incorporate more or less input elements.For example, the display 405 may not be a touch screen in someembodiments. In yet another alternative embodiment, the mobile computingdevice 400 is a portable phone system, such as a cellular phone. Themobile computing device 400 may also include an optional keypad 435.Optional keypad 435 may be a physical keypad or a “soft” keypadgenerated on the touch screen display. In various embodiments, theoutput elements include the display 405 for showing a graphical userinterface (GUI), a visual indicator 420 (e.g., a light emitting diode),and/or an audio transducer 425 (e.g., a speaker). In some aspects, themobile computing device 400 incorporates a vibration transducer forproviding the user with tactile feedback. In yet another aspect, themobile computing device 400 incorporates input and/or output ports, suchas an audio input (e.g., a microphone jack), an audio output (e.g., aheadphone jack), and a video output (e.g., a HDMI port) for sendingsignals to or receiving signals from an external device.

FIG. 4B is a block diagram illustrating the architecture of one aspectof a mobile computing device. That is, the mobile computing device 400can incorporate a system (e.g., an architecture) 402 to implement someaspects. In one embodiment, the system 402 is implemented as a “smartphone” capable of running one or more applications (e.g., browser,e-mail, calendaring, contact managers, messaging clients, games, andmedia clients/players). In some aspects, the system 402 is integrated asa computing device, such as an integrated personal digital assistant(PDA) and wireless phone.

One or more application programs 466 may be loaded into the memory 462and run on or in association with the operating system 464. Examples ofthe application programs include phone dialer programs, e-mail programs,personal information management (PIM) programs, word processingprograms, spreadsheet programs, Internet browser programs, messagingprograms, and so forth. The system 402 also includes a non-volatilestorage area 468 within the memory 462. The non-volatile storage area468 may be used to store persistent information that should not be lostif the system 402 is powered down. The application programs 466 may useand store information in the non-volatile storage area 468, such ase-mail or other messages used by an e-mail application, and the like. Asynchronization application (not shown) also resides on the system 402and is programmed to interact with a corresponding synchronizationapplication resident on a host computer to keep the information storedin the non-volatile storage area 468 synchronized with correspondinginformation stored at the host computer. As should be appreciated, otherapplications may be loaded into the memory 462 and run on the mobilecomputing device 400 described herein (e.g., search engine, extractormodule, relevancy ranking module, answer scoring module, etc.).

The system 402 has a power supply 470, which may be implemented as oneor more batteries. The power supply 470 might further include anexternal power source, such as an AC adapter or a powered docking cradlethat supplements or recharges the batteries.

The system 402 may also include a radio interface layer 472 thatperforms the function of transmitting and receiving radio frequencycommunications. The radio interface layer 472 facilitates wirelessconnectivity between the system 402 and the “outside world,” via acommunications carrier or service provider. Transmissions to and fromthe radio interface layer 472 are conducted under control of theoperating system 464. In other words, communications received by theradio interface layer 472 may be disseminated to the applicationprograms 466 via the operating system 464, and vice versa.

The visual indicator 420 may be used to provide visual notifications,and/or an audio interface 474 may be used for producing audiblenotifications via the audio transducer 425. In the illustratedembodiment, the visual indicator 420 is a light emitting diode (LED) andthe audio transducer 425 is a speaker. These devices may be directlycoupled to the power supply 470 so that when activated, they remain onfor a duration dictated by the notification mechanism even though theprocessor 460 and other components might shut down for conservingbattery power. The LED may be programmed to remain on indefinitely untilthe user takes action to indicate the powered-on status of the device.The audio interface 474 is used to provide audible signals to andreceive audible signals from the user. For example, in addition to beingcoupled to the audio transducer 425, the audio interface 474 may also becoupled to a microphone to receive audible input, such as to facilitatea telephone conversation. In accordance with embodiments of the presentdisclosure, the microphone may also serve as an audio sensor tofacilitate control of notifications, as will be described below. Thesystem 402 may further include a video interface 476 that enables anoperation of an on-board camera 430 to record still images, videostream, and the like.

A mobile computing device 400 implementing the system 402 may haveadditional features or functionality. For example, the mobile computingdevice 400 may also include additional data storage devices (removableand/or non-removable) such as, magnetic disks, optical disks, or tape.Such additional storage is illustrated in FIG. 4B by the non-volatilestorage area 468.

Data/information generated or captured by the mobile computing device400 and stored via the system 402 may be stored locally on the mobilecomputing device 400, as described above, or the data may be stored onany number of storage media that may be accessed by the device via theradio interface layer 472 or via a wired connection between the mobilecomputing device 400 and a separate computing device associated with themobile computing device 400, for example, a server computer in adistributed computing network, such as the Internet. As should beappreciated such data/information may be accessed via the mobilecomputing device 400 via the radio interface layer 472 or via adistributed computing network. Similarly, such data/information may bereadily transferred between computing devices for storage and useaccording to well-known data/information transfer and storage means,including electronic mail and collaborative data/information sharingsystems.

FIG. 5 illustrates one aspect of the architecture of a system forprocessing data received at a computing system from a remote source,such as a personal computer 504, tablet computing device 506, or mobilecomputing device 508, as described above. Content displayed at serverdevice 502 may be stored in different communication channels or otherstorage types. For example, various documents may be stored using adirectory service 522, a web portal 524, a mailbox service 526, aninstant messaging store 528, or a social networking site 530.Parallel-hierarchical machine comprehension process 521 may be employedby a client that communicates with server device 502, and/orparallel-hierarchical machine comprehension process 520 may be employedby server device 502. The server device 502 may provide data to and froma client computing device such as a personal computer 504, a tabletcomputing device 506 and/or a mobile computing device 508 (e.g., a smartphone) through a network 515. By way of example, the computer systemdescribed above may be embodied in a personal computer 504, a tabletcomputing device 506 and/or a mobile computing device 508 (e.g., a smartphone). Any of these embodiments of the computing devices may obtaincontent from the store 516, in addition to receiving graphical datauseable to be either pre-processed at a graphic-originating system, orpost-processed at a receiving computing system.

FIG. 5 illustrates an exemplary tablet computing device 500 that mayexecute one or more aspects disclosed herein. In addition, the aspectsand functionalities described herein may operate over distributedsystems (e.g., cloud-based computing systems), where applicationfunctionality, memory, data storage and retrieval and various processingfunctions may be operated remotely from each other over a distributedcomputing network, such as the Internet or an intranet. User interfacesand information of various types may be displayed via on-board computingdevice displays or via remote display units associated with one or morecomputing devices. For example user interfaces and information ofvarious types may be displayed and interacted with on a wall surfaceonto which user interfaces and information of various types areprojected. Interaction with the multitude of computing systems withwhich embodiments of the invention may be practiced include, keystrokeentry, touch screen entry, voice or other audio entry, gesture entrywhere an associated computing device is equipped with detection (e.g.,camera) functionality for capturing and interpreting user gestures forcontrolling the functionality of the computing device, and the like.

FIG. 6 illustrates an exemplary tablet computing device 600 that mayexecute one or more aspects disclosed herein. In addition, the aspectsand functionalities described herein may operate over distributedsystems (e.g., cloud-based computing systems), where applicationfunctionality, memory, data storage and retrieval and various processingfunctions may be operated remotely from each other over a distributedcomputing network, such as the Internet or an intranet. User interfacesand information of various types may be displayed via on-board computingdevice displays or via remote display units associated with one or morecomputing devices. For example, user interfaces and information ofvarious types may be displayed and interacted with on a wall surfaceonto which user interfaces and information of various types areprojected. Interaction with the multitude of computing systems withwhich embodiments of the invention may be practiced include, keystrokeentry, touch screen entry, voice or other audio entry, gesture entrywhere an associated computing device is equipped with detection (e.g.,camera) functionality for capturing and interpreting user gestures forcontrolling the functionality of the computing device, and the like.

The Parallel-Hierarchical aspects disclosed herein does not requirecomplex language or sequence modeling. Nevertheless, the model hasproven quite effective for machine comprehension tasks using small datasets, for example the MCTest and other data sets. Although the so-called“multi” questions (putatively) require some limited reasoning, the modelis able to handle them reasonably well just by stringing importantsentences together. Thus, the model imitates reasoning with a heuristic.

However, it may be that human-level language processing can be factoredinto separate processes of comprehension and reasoning. If so, theParallel-Hierarchical model is a good start on the former. Indeed, themethod exclusively was trained on single questions then its resultsbecome even more impressive: it can achieve a test accuracy of 79.1% onMCTest-500. The Parallel-Hierarchical aspects for machine comprehensiondisclosed herein, achieves state-of-the-art results, outperformingseveral feature-based and learning-based approaches. Work on the modelhas emphasized the following: Language is naturally hierarchical, andgood comprehension of language relies on hierarchical levels ofunderstanding; exogenous attention (the trainable word weights) may bebroadly helpful for NLP; the training wheels approach, that is,initializing neural networks to perform sensible heuristics, appearshelpful for small datasets; and reasoning over language is challenging,but easily simulated in some cases.

Aspects of the present disclosure, for example, are described above withreference to block diagrams and/or operational illustrations of methods,systems, and computer program products according to aspects of thedisclosure. The functions/acts noted in the blocks may occur out of theorder as shown in any flowchart. For example, two blocks shown insuccession may in fact be executed substantially concurrently or theblocks may sometimes be executed in the reverse order, depending uponthe functionality/acts involved.

The description and illustration of one or more aspects provided in thisapplication are not intended to limit or restrict the scope of thedisclosure as claimed in any way. The aspects, examples, and detailsprovided in this application are considered sufficient to conveypossession and enable others to make and use the best mode of claimeddisclosure. The claimed disclosure should not be construed as beinglimited to any aspect, example, or detail provided in this application.Regardless of whether shown and described in combination or separately,the various features (both structural and methodological) are intendedto be selectively included or omitted to produce an embodiment with aparticular set of features. Having been provided with the descriptionand illustration of the present application, one skilled in the art mayenvision variations, modifications, and alternate aspects falling withinthe spirit of the broader aspects of the general inventive conceptembodied in this application that do not depart from the broader scopeof the claimed disclosure.

What is claimed is:
 1. A system comprising: at least one processor; andmemory storing instructions that, when executed by the at least oneprocessor, perform a set of operations comprising: receiving input text,wherein the input text comprises text data, a question, and an answercandidate for the question based on the text data; converting the inputtext into a word-by-word representation based at least on a word-by-wordsentential match between a combination of the natural language questionand the answer candidate and the text data; converting the input textinto a semantic representation of the input text based at least on asemantic match between the combination of the question and the answercandidate and the text data; concurrently processing: generating a firstresult based on analyzing the word-by-word representation using one ormore word-by-word processes, wherein each word-by-word process generatesa first matching score indicating a degree of the word-by-wordsentential match; and generating a second result based on analyzing thesemantic representation using semantic processes, wherein each semanticprocess generates a second matching score indicating a degree of thesemantic match; combining the first result and the second result;determining, based on the combined first and second results, a topresult, wherein the top result includes the combination of the questionand the answer candidate corresponding to the highest score among a setof matching scores including the first matching score and the secondmatching score; and providing the top result.
 2. The system of claim 1,wherein each word-by-word processes use a multilayer perceptron neuralnetwork including the natural language question and the answercandidate, wherein the semantic processes use a multilayer perceptronplus sum neural network based on semantics of the input text, andwherein the first result and the second result are distinct.
 3. Thesystem of claim 1, wherein the input text comprises a natural languagequestion.
 4. The system of claim 1, wherein the concurrently processingcomprises a semantic analysis of the input text and a word-by-wordanalysis of the input text.
 5. The system of claim 4, wherein thesemantic analysis includes comparing a hypothesis to sentences in thetext data.
 6. The system of claim 5, wherein the hypothesis includes acombination of at least a portion of the question with at least aportion of the answer candidate.
 7. The system of claim 4, wherein theword-by-word process comprises at least one of: a sentential process; asliding window sequential process; and a dependency sliding windowdependency process.
 8. The system of claim 7, wherein the sliding windowsequential process scans over words of the text data as one continuoussequence.
 9. The system of claim 7, wherein the dependency slidingwindow dependency process comprises: constructing a dependency graph fora sentence in the text data; reordering words in the sentence based atleast in part on the dependency graph to generate a reordered sentence;and scans over words of the reordered sentence.
 10. The system of claim1, wherein the concurrently processing uses at least a multilayerperceptron neural network.
 11. A method comprising: receiving inputtext, wherein the input text comprises text data, a natural languagequestion, and an answer candidate for the natural language questionbased on the text data; converting the input text into a word-by-wordrepresentation based at least on a word-by-word sentential match betweena combination of the natural language question and the answer candidateand the text data; converting the input text into a semanticrepresentation of the input text based at least on a semantic matchbetween the combination of the natural language question and the answercandidate and the text data; concurrently processing: generating a firstresult based on analyzing the word-by-word representation using one ormore word-by-word processes, wherein each word-by-word process generatesa first matching score indicating a degree of the word-by-wordsentential match; and generating a second result based on analyzing thesemantic representation using semantic processes, wherein each semanticprocess generates a second matching score indicating a degree of thesemantic match; combining the first result and the second resultdetermining, based on the combined first and second results, a topresult, wherein the top result includes the combination of the naturallanguage question and the answer candidate corresponding to the highestscore among a set of matching scores including the first matching scoreand the second matching score; and providing the top result.
 12. Themethod of claim 11, wherein each word-by-word processes use a multilayerperceptron neural network including the natural language question andthe answer candidate, wherein the semantic processes use a multilayerperceptron plus sum neural network based on semantics of the input textwith a summation using a weight associated with a word, and wherein thefirst result and the second result are distinct.
 13. The method of claim12, wherein the semantic processes include comparing a hypothesis tosentences in the text data.
 14. The method of claim 13, wherein thehypothesis includes a combination of at least a portion of the naturallanguage question with at least a portion of the answer candidate. 15.The method of claim 14, wherein the hypothesis is compared to the textdata using cosine similarity.
 16. The method of claim 11, wherein theone or more word-by-word process comprises at least one of: a sententialprocess; a sliding window sequential process; and a dependency slidingwindow dependency process.
 17. The method of claim 16, wherein thesliding window sequential process scans over words of the text data asone continuous sequence.
 18. The method of claim 16, wherein thedependency sliding window dependency process comprises: constructing adependency graph for a sentence in the text data; reordering words inthe sentence based at least in part on the dependency graph to generatea reordered sentence; and scans over words of the reordered sentence.19. A computer storage medium comprising computer executableinstructions that, when executed by at least one processor, executes amethod comprising: receiving input text, wherein the input textcomprises text data, a natural language question, and an answercandidate for the natural language question based on the text data;converting the input text into a word-by-word representation based atleast on a word-by-word sentential match between a combination of thenatural language question and the answer candidate and the text data;converting the input text into a semantic representation of the inputtext based at least on a semantic match between the combination of thenatural language question and the answer candidate and the text data;concurrently processing: generating a first result based on analyzingthe word-by-word representation using one or more word-by-wordprocesses, wherein each word-by-word process generates a first matchingscore indicating a degree of the word-by-word sentential match; andgenerating a second result based on analyzing the semanticrepresentation using semantic processes, wherein each semantic processgenerates a second matching score indicating a degree of the semanticmatch; combining the first result and the second result; determining,based on the combined first and second results, a top result, whereinthe top result includes the combination of the natural language questionand the answer candidate corresponding to the highest score among a setof matching scores including the first matching score and the secondmatching score; and providing the top result.
 20. The computer storagemedium of claim 19, wherein each word-by-word processes use a multilayerperceptron neural network including the natural language question andthe answer candidate, wherein the semantic processes use a multilayerperceptron plus sum neural network based on semantics of the input textbased on semantics of the input text with a summation using a weightassociated with a word, and wherein the first result and the secondresult are distinct.